Knowledge distillation for BERT unsupervised domain adaptation

A pre-trained language model, BERT, has brought significant performance improvements across a range of natural processing tasks. Since the model is trained on large corpus diverse topics, it shows robust for domain shift problems in which data distributions at training (source data) and testing (target differ while sharing similarities. Despite its great compared to previous models, still suffers from degradation due shifts. To mitigate such problems, we propose simple but effective unsupervised adaptation method, adversarial with distillation (AAD), combines discriminative (ADDA) framework knowledge distillation. We evaluate our approach task cross-domain sentiment classification 30 pairs, advancing state-of-the-art text classification.